Remove StructuralVariant and StructuralVariantType, add names field to Variant #1131

heuermh · 2016-08-25T19:19:21Z

Still left to do (in a later pull request) is to populate the somatic field from VCF INFO attributes.

AmplabJenkins · 2016-08-25T19:23:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1425/

Build result: FAILURE

GitHub pull request #1131 of commit e8774e9 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1131/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains e3194f8 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/merge^{commit} # timeout=10Checking out Revision e3194f8 (origin/pr/1131/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f e3194f87b2dda6dc4757303228f43d1099d005d5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2016-08-25T19:25:10Z

This won't pass CI until bigdatagenomics/bdg-formats#103 is merged and the new snapshot version of bdg-formats is available.

heuermh · 2016-08-26T16:59:45Z

Something is git-wrong with this branch.

It is supposed to have the commit from #1126 in it (bf886c9) but instead has the same commit with a different hash (ea53e2d).

AmplabJenkins · 2016-08-26T17:03:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1430/

Build result: FAILURE

GitHub pull request #1131 of commit 5f55d86 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1131/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains a28e8fe # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/merge^{commit} # timeout=10Checking out Revision a28e8fe (origin/pr/1131/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f a28e8fe3a08c123682bd4ce66ea800741137fffbFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2016-08-26T17:06:27Z

How'd you pull the commit over from #1126? Also, when I merged #1126 into the topic branch, I did github's squash-and-merge, which (unfortunately) triggers an update to the git commit hash.

heuermh · 2016-08-26T17:53:36Z

I cherry picked it over originally, then l did a rebase here against the topic branch.

fnothaft · 2016-08-27T03:43:06Z

If you cherry pick the commit over, I wouldn't expect the hash to be the same anyways.

jpdna · 2016-08-27T17:35:44Z

+1
I round tripped a VCF with vcf2adam then adam2vcf and the names (rs_ids) are present in the VCF as advertised.

fnothaft · 2016-08-27T17:50:23Z

adam-core/src/main/scala/org/bdgenomics/adam/converters/VariantContextConverter.scala

+   * @return Returns an Option wrapping an array of names split from the htsjdk
+   *    variant context ID field
+   */
+  private def splitIds(vc: HtsjdkVariantContext): Option[java.util.List[String]] = {


Can you write some unit tests for this? Ideally, we'd have tests for just this function with a VC with no ID, one ID, more than one ID, and an end-to-end VCF-with-IDs test.

Can you write some unit tests for this?

As in me Justin? I'd be happy to work on the unit tests - would be good for me. But if @heuermh wants to as his PR he can have first dibs. Just let me know what you prefer Michael - and if you want me to do it, should it be a separate PR or do you want me to PR against your PR Michael?

Oh, sorry for the confusion! My comment was directed at @heuermh, unless he'd like you to work on it, of course!

No worries, I got it!

fnothaft · 2016-08-27T17:51:00Z

LGTM other than test nit!

heuermh · 2016-08-29T17:07:07Z

Added unit tests, rebased, and force pushed. Sorry about the conflicts, not sure how to resolve them.

AmplabJenkins · 2016-08-29T17:11:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1433/

Build result: FAILURE

GitHub pull request #1131 of commit ed3c420.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse ed3c420^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains ed3c420 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision ed3c420 (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ed3c4205748cb4025d60657d1ed65c632ab5e432First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2016-08-29T18:18:13Z

Should I register mutable.ArrayBuffer or modify the unit tests (where I use Seq())?

- can write, then read in .vcf file *** FAILED ***
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4, localhost): com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.collection.mutable.ArrayBuffer
Note: To register this class use: kryo.register(scala.collection.mutable.ArrayBuffer.class);

fnothaft · 2016-08-29T18:26:43Z

I'd probably register it.

AmplabJenkins · 2016-08-29T19:10:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1434/

Build result: FAILURE

GitHub pull request #1131 of commit 385cb9e.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse 385cb9e^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 385cb9e # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision 385cb9e (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 385cb9e > /home/jenkins/git2/bin/git rev-list ed3c420 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2016-08-29T19:18:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1435/

Build result: FAILURE

GitHub pull request #1131 of commit 34877cf.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse 34877cf^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 34877cf # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision 34877cf (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 34877cf > /home/jenkins/git2/bin/git rev-list 385cb9e # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2016-08-29T19:22:34Z

Jenkins, retest this please.

AmplabJenkins · 2016-08-29T19:25:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1436/

Build result: FAILURE

GitHub pull request #1131 of commit 34877cf.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse 34877cf^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 34877cf # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision 34877cf (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 34877cf > /home/jenkins/git2/bin/git rev-list 34877cf # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

…o Variant

AmplabJenkins · 2016-08-29T19:36:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1437/

Build result: FAILURE

GitHub pull request #1131 of commit b56b77c.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse b56b77c^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b56b77c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision b56b77c (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b56b77c > /home/jenkins/git2/bin/git rev-list 34877cf # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2016-08-29T19:43:32Z

All right, this is starting to piss me off. Why can't I replicate this test failure locally?

heuermh · 2016-08-30T02:44:11Z

@fnothaft sorry, I'm not sure what to do to fix this one.

"spark.kryo.registrationRequired" -> "true" in ADAMFunSuite but I don't see these test failures locally.

heuermh · 2016-08-30T23:01:01Z

Jenkins, retest this please.

AmplabJenkins · 2016-08-30T23:19:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1439/
Test PASSed.

heuermh · 2016-08-30T23:27:54Z

WTF! Please merge this before Jenkins changes his mind.

fnothaft · 2016-08-31T23:21:33Z

Merged into upgrade-to-bdg-formats-0.10.0 as 97d9290. I squashed down a spurious cleanup commit and rebased upgrade-to-bdg-formats-0.10.0 on master. Thanks @heuermh!

heuermh · 2016-08-31T23:22:57Z

Great, thank you!

fnothaft and others added 2 commits August 24, 2016 12:48

Bump to bdg-formats 0.10.0-SNAPSHOT.

d86f258

Update for bdg-formats code style changes

5e0d8b9

heuermh mentioned this pull request Aug 25, 2016

Variant dbsnp rs id tracking in vcf2adam and ADAM2Vcf #1103

Closed

heuermh mentioned this pull request Aug 26, 2016

Load SV type info field - need for allele uniquness #1134

Closed

heuermh force-pushed the formats-issue-102 branch from e8774e9 to 5f55d86 Compare August 26, 2016 16:56

fnothaft reviewed Aug 27, 2016
View reviewed changes

heuermh force-pushed the formats-issue-102 branch from 5f55d86 to ed3c420 Compare August 29, 2016 17:05

heuermh force-pushed the formats-issue-102 branch from ed3c420 to 385cb9e Compare August 29, 2016 19:07

heuermh force-pushed the formats-issue-102 branch from 385cb9e to 34877cf Compare August 29, 2016 19:15

Remove StructuralVariant and StructuralVariantType, add names field t…

b56b77c

…o Variant

heuermh force-pushed the formats-issue-102 branch from 34877cf to b56b77c Compare August 29, 2016 19:30

fnothaft force-pushed the upgrade-to-bdg-formats-0.10.0 branch from 65adcf6 to 97d9290 Compare August 31, 2016 23:20

fnothaft closed this Aug 31, 2016

heuermh deleted the formats-issue-102 branch August 31, 2016 23:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove StructuralVariant and StructuralVariantType, add names field to Variant #1131

Remove StructuralVariant and StructuralVariantType, add names field to Variant #1131

heuermh commented Aug 25, 2016

AmplabJenkins commented Aug 25, 2016

heuermh commented Aug 25, 2016

heuermh commented Aug 26, 2016

AmplabJenkins commented Aug 26, 2016

fnothaft commented Aug 26, 2016

heuermh commented Aug 26, 2016

fnothaft commented Aug 27, 2016

jpdna commented Aug 27, 2016

fnothaft Aug 27, 2016

jpdna Aug 27, 2016

fnothaft Aug 27, 2016

heuermh Aug 27, 2016

fnothaft commented Aug 27, 2016

heuermh commented Aug 29, 2016

AmplabJenkins commented Aug 29, 2016

heuermh commented Aug 29, 2016

fnothaft commented Aug 29, 2016

AmplabJenkins commented Aug 29, 2016

AmplabJenkins commented Aug 29, 2016

heuermh commented Aug 29, 2016

AmplabJenkins commented Aug 29, 2016

AmplabJenkins commented Aug 29, 2016

heuermh commented Aug 29, 2016

heuermh commented Aug 30, 2016

heuermh commented Aug 30, 2016

AmplabJenkins commented Aug 30, 2016

heuermh commented Aug 30, 2016

fnothaft commented Aug 31, 2016

heuermh commented Aug 31, 2016

Remove StructuralVariant and StructuralVariantType, add names field to Variant #1131

Remove StructuralVariant and StructuralVariantType, add names field to Variant #1131

Conversation

heuermh commented Aug 25, 2016

AmplabJenkins commented Aug 25, 2016

Build result: FAILURE

heuermh commented Aug 25, 2016

heuermh commented Aug 26, 2016

AmplabJenkins commented Aug 26, 2016

Build result: FAILURE

fnothaft commented Aug 26, 2016

heuermh commented Aug 26, 2016

fnothaft commented Aug 27, 2016

jpdna commented Aug 27, 2016

fnothaft Aug 27, 2016

Choose a reason for hiding this comment

jpdna Aug 27, 2016

Choose a reason for hiding this comment

fnothaft Aug 27, 2016

Choose a reason for hiding this comment

heuermh Aug 27, 2016

Choose a reason for hiding this comment

fnothaft commented Aug 27, 2016

heuermh commented Aug 29, 2016

AmplabJenkins commented Aug 29, 2016

Build result: FAILURE

heuermh commented Aug 29, 2016

fnothaft commented Aug 29, 2016

AmplabJenkins commented Aug 29, 2016

Build result: FAILURE

AmplabJenkins commented Aug 29, 2016

Build result: FAILURE

heuermh commented Aug 29, 2016

AmplabJenkins commented Aug 29, 2016

Build result: FAILURE

AmplabJenkins commented Aug 29, 2016

Build result: FAILURE

heuermh commented Aug 29, 2016

heuermh commented Aug 30, 2016

heuermh commented Aug 30, 2016

AmplabJenkins commented Aug 30, 2016

heuermh commented Aug 30, 2016

fnothaft commented Aug 31, 2016

heuermh commented Aug 31, 2016